Speech Enhancement In Time Domain Research Articles

Time domain speech enhancement with CNN and time-attention transformer

Speech enhancement in the time domain involves improving the quality and intelligibility of noisy speech by processing the waveform directly without the need for explicit feature extraction or domain transformation. Deep learning is a powerful approach for time domain speech enhancement, offering significant improvements over traditional techniques. Formulating a resource-efficient deep neural model in the time domain without ignoring the contextual information and detailed features of input speech is still a vital challenge. To address this challenge, this study proposes a speech enhancement model using 1D-time domain dilated residual blocks in the convolutional encoder-decoder framework. Further, this study integrates a time-attention transformer (TAT) bottleneck between the encoder-decoder. The TAT model extends the transformer architecture by incorporating a time-attention mechanism, which enables the model to selectively attend to different segments of the speech signal over time. This allows the model to effectively capture long-term dependencies in the speech and learn to recognize important features. The experimental results indicate that the proposed speech enhancement outperforms the recent deep neural networks (DNNs) and substantially improves the intelligibility and quality of noisy speech. With the WSJ0 SI-84 database, the proposed SE improves the STOI and PESQ by 21.51% and 1.14 over noisy speech.

New modified Bat algorithm for blind speech enhancement in time domain

We address the speech enhancement problem for dual convolutif mixed channel by viewing it in a blind separation sources setting. One widely used technique to separate mixed signals is to apply adaptive filtering, the challenge is to identify an unknown finite impulse response. traditionally we apply a gradient-based algorithm to adapt filter coefficients. However, such algorithm often suffers from premature convergence ,when using large filters and non-stationary inputs ,  leading to the so-called local minimum problem , which affects the quality of  enhanced signals significatively .one alternative to overcome this problem is to apply a population-based metaheuristic algorithms in which filter coefficients are adapted iteratively  by minimizing a cost function .But even with this metaheuristic based solution, local minimum problem at large filters still exists. In order to avoid local minima and improve the chance to reach the global solution, we propose in this paper, a novel algorithm called a modified bat algorithm to render the search process efficiently by enhancing its capability of exploration and exploitation. Several experiments under different noise types are carried out using our proposed modified bat algorithm in comparison with some of the popular state-of-the-art algorithms. The enhanced signals obtained by each algorithm at the separation process outputs, show good behavior and superiority of our proposed algorithm. In terms of system misalignment, as well as a segmental signal-to-noise ratio.

Shuffle Attention U-Net for Speech Enhancement in Time Domain

Over the past 10 years, deep learning has enabled significant advancements in the improvement of noisy speech. In an end-to-end speech enhancement, the deep neural networks transform a noisy speech signal to a clean speech signal in the time domain directly without any conversion or estimation of mask. Recently, the U-Net-based models achieved good enhancement performance. Despite this, some of them may neglect context-related information and detailed features of input speech in case of ordinary convolution. To address the above issues, recent studies have upgraded the performance of the model by adding various network modules such as attention mechanisms, long and short-term memory (LSTM). In this work, we propose a new U-Net-based speech enhancement model using a novel lightweight and efficient Shuffle Attention (SA), Gated Recurrent Unit (GRU), residual blocks with dilated convolutions. Residual block will be followed by a multi-scale convolution block (MSCB). The proposed hybrid structure enables the temporal context aggregation in time domain. The advantage of shuffle attention mechanism is that the channel and spatial attention are carried out simultaneously for each sub-feature in order to prevent potential noises while also highlighting the proper semantic feature areas by combining the same features from all locations. MSCB is employed for extracting rich temporal features. To represent the correlation between neighboring noisy speech frames, a two Layer GRU is added in the bottleneck of U-Net. The experimental findings demonstrate that the proposed model outperformed the other existing models in terms of short-time objective intelligibility (STOI), and perceptual evaluation of the speech quality (PESQ).

Convolutional Transformer based Local and Global Feature Learning for Speech Enhancement

Speech enhancement (SE) is an important method for improving speech quality and intelligibility in noisy environments where received speech is severely distorted by noise. An efficient speech enhancement system relies on accurately modelling the long-term dependencies of noisy speech. Deep learning has greatly benefited by the use of transformers where long-term dependencies can be modelled more efficiently with multi-head attention (MHA) by using sequence similarity. Transformers frequently outperform recurrent neural network (RNN) and convolutional neural network (CNN) models in many tasks while utilizing parallel processing. In this paper we proposed a two-stage convolutional transformer for speech enhancement in time domain. The transformer considers global information as well as parallel computing, resulting in a reduction of long-term noise. In the proposed work unlike two-stage transformer neural network (TSTNN) different transformer structures for intra and inter transformers are used for extracting the local as well as global features of noisy speech. Moreover, a CNN module is added to the transformer so that short-term noise can be reduced more effectively, based on the ability of CNN to extract local information. The experimental findings demonstrate that the proposed model outperformed the other existing models in terms of STOI (short-time objective intelligibility), and PESQ (perceptual evaluation of the speech quality).

Optimizing the Perceptual Quality of Time-Domain Speech Enhancement with Reinforcement Learning

In neural speech enhancement, a mismatch exists between the training objective, i.e., Mean-Square Error (MSE), and perceptual quality evaluation metrics, i.e., perceptual evaluation of speech quality and short-time objective intelligibility. We propose a novel reinforcement learning algorithm and network architecture, which incorporate a non-differentiable perceptual quality evaluation metric into the objective function using a dynamic filter module. Unlike the traditional dynamic filter implementation that directly generates a convolution kernel, we use a filter generation agent to predict the probability density function of a multivariate Gaussian distribution, from which we sample the convolution kernel. Experimental results show that the proposed reinforcement learning method clearly improves the perceptual quality over other supervised learning methods with the MSE objective function.

Multi-modal speech enhancement with bone-conducted speech in time domain

Bone-conducted (BC) speech captures speech signals based on the vibrations of a speaker’s skull. It is thus not affected by noise sources from environments and hence exhibits better noise-resistance capabilities than air-conducted (AC) speech. Although the quality and intelligibility of the BC speech degrade due to the nature of the solid vibration, BC speech can be utilized as an auxiliary source to jointly improve the performance of speech enhancement. In this paper, we propose an end-to-end multi-modal model for time-domain speech enhancement at low signal-to-noise ratios. The model utilizes both noisy AC speech and synchronized BC speech as the input. It takes an encoder-decoder architecture, where an involution network is used to estimate the mask of clean speech component, and the mask is then applied to remove the noise component. We compared the proposed method with several state-of-the-art multi-modal and single-modal methods on an air- and bone-conducted multi-modal corpus. Experimental results demonstrate that the proposed approach outperforms the comparison methods in terms of the speech quality and intelligibility of the enhanced speech. When applied to speech recognition, the enhanced speech significantly reduces the error rate.

Short-term uncleaned signal to noise threshold ratio based end-to-end time domain speech enhancement in digital hearing aids

This paper presents the improvements in the combined solution for the noise estimation and the speech enhancement in digital hearing aids in time domain. This study focuses on the single channel statistical temporal speech enhancement using adaptive Wiener filtering. In this technique, the noise is updated based on the short-term uncleaned signal to noise threshold ratio (ST-USNTR) of the frame. It works best if and only if the back ground noise level is low compared to that of speech of interest. We considered the time domain algorithms in order to consider the time varying nature of speech signal. The performance of the proposed algorithm is evaluated for speech signal with seven ty pes of noises and three signal to noise ratios (SNR) levels in each type of noise. From the results, it is clear that the basic level of adaptive speech enhancement is obtained using statistical parameters of noisy speech without the need for reference input.

Hybrid Dilated and Recursive Recurrent Convolution Network for Time-Domain Speech Enhancement

In this paper, we propose a fully convolutional neural network based on recursive recurrent convolution for monaural speech enhancement in the time domain. The proposed network is an encoder-decoder structure using a series of hybrid dilated modules (HDM). The encoder creates low-dimensional features of a noisy input frame. In the HDM, the dilated convolution is used to expand the receptive field of the network model. In contrast, the standard convolution is used to make up for the under-utilized local information of the dilated convolution. The decoder is used to reconstruct enhanced frames. The recursive recurrent convolutional network uses GRU to solve the problem of multiple training parameters and complex structures. State-of-the-art results are achieved on two commonly used speech datasets.

Self-attending RNN for Speech Enhancement to Improve Cross-corpus Generalization.

Deep neural networks (DNNs) represent the mainstream methodology for supervised speech enhancement, primarily due to their capability to model complex functions using hierarchical representations. However, a recent study revealed that DNNs trained on a single corpus fail to generalize to untrained corpora, especially in low signal-to-noise ratio (SNR) conditions. Developing a noise, speaker, and corpus independent speech enhancement algorithm is essential for real-world applications. In this study, we propose a self-attending recurrent neural network (SARNN) for time-domain speech enhancement to improve cross-corpus generalization. SARNN comprises of recurrent neural networks (RNNs) augmented with self-attention blocks and feedforward blocks. We evaluate SARNN on different corpora with nonstationary noises in low SNR conditions. Experimental results demonstrate that SARNN substantially outperforms competitive approaches to time-domain speech enhancement, such as RNNs and dual-path SARNNs. Additionally, we report an important finding that the two popular approaches to speech enhancement: complex spectral mapping and time-domain enhancement, obtain similar results for RNN and SARNN with large-scale training. We also provide a challenging subset of the test set used in this study for evaluating future algorithms and facilitating direct comparisons.

Improving the Intelligibility of Speech for Simulated Electric and Acoustic Stimulation Using Fully Convolutional Neural Networks.

Combined electric and acoustic stimulation (EAS) has demonstrated better speech recognition than conventional cochlear implant (CI) and yielded satisfactory performance under quiet conditions. However, when noise signals are involved, both the electric signal and the acoustic signal may be distorted, thereby resulting in poor recognition performance. To suppress noise effects, speech enhancement (SE) is a necessary unit in EAS devices. Recently, a time-domain speech enhancement algorithm based on the fully convolutional neural networks (FCN) with a short-time objective intelligibility (STOI)-based objective function (termed FCN(S) in short) has received increasing attention due to its simple structure and effectiveness of restoring clean speech signals from noisy counterparts. With evidence showing the benefits of FCN(S) for normal speech, this study sets out to assess its ability to improve the intelligibility of EAS simulated speech. Objective evaluations and listening tests were conducted to examine the performance of FCN(S) in improving the speech intelligibility of normal and vocoded speech in noisy environments. The experimental results show that, compared with the traditional minimum-mean square-error SE method and the deep denoising autoencoder SE method, FCN(S) can obtain better gain in the speech intelligibility for normal as well as vocoded speech. This study, being the first to evaluate deep learning SE approaches for EAS, confirms that FCN(S) is an effective SE approach that may potentially be integrated into an EAS processor to benefit users in noisy environments.

Dense CNN with Self-Attention for Time-Domain Speech Enhancement.

Speech enhancement in the time domain is becoming increasingly popular in recent years, due to its capability to jointly enhance both the magnitude and the phase of speech. In this work, we propose a dense convolutional network (DCN) with self-attention for speech enhancement in the time domain. DCN is an encoder and decoder based architecture with skip connections. Each layer in the encoder and the decoder comprises a dense block and an attention module. Dense blocks and attention modules help in feature extraction using a combination of feature reuse, increased network depth, and maximum context aggregation. Furthermore, we reveal previously unknown problems with a loss based on the spectral magnitude of enhanced speech. To alleviate these problems, we propose a novel loss based on magnitudes of enhanced speech and a predicted noise. Even though the proposed loss is based on magnitudes only, a constraint imposed by noise prediction ensures that the loss enhances both magnitude and phase. Experimental results demonstrate that DCN trained with the proposed loss substantially outperforms other state-of-the-art approaches to causal and non-causal speech enhancement.

FLGCNN: A novel fully convolutional neural network for end-to-end monaural speech enhancement with utterance-based objective functions

This paper proposes a novel fully convolutional neural network (FCN) called FLGCNN to address the end-to-end speech enhancement in time domain. The proposed FLGCNN is mainly built on encoder and decoder, while the extra convolutional-based short-time Fourier transform (CSTFT) layer and inverse STFT (CISTFT) layer are added to emulate the forward and inverse STFT operations. These layers aim to integrate the frequency-domain knowledge into the proposed model since the underlying phonetic information of speech is presented more clearly by time–frequency (T-F) representations. In addition, the encoder and decoder are constructed by the gated convolutional layers so that the proposed model can better control the information passed on in the hierarchy. Besides, motivated by the popular temporal convolutional neural network (TCNN), the temporal convolutional module (TCM) which is efficient in modeling the long-term dependencies of speech signal is inserted between encoder and decoder. We also optimize the proposed model with different utterance-based objective functions to exploit the impact of loss functions on performance, because the entire framework can realize the end-to-end speech enhancement. Experimental results have demonstrated that the proposed model consistently gives better performance improvement than the other competitive methods of speech enhancement.

On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement

Many deep learning-based speech enhancement algorithms are designed to minimize the mean-square error (MSE) in some transform domain between a predicted and a target speech signal. However, optimizing for MSE does not necessarily guarantee high speech quality or intelligibility, which is the ultimate goal of many speech enhancement algorithms. Additionally, only little is known about the impact of the loss function on the emerging class of time-domain deep learning-based speech enhancement systems. We study how popular loss functions influence the performance of deep learning-based speech enhancement systems. First, we demonstrate that perceptually inspired loss functions might be advantageous if the receiver is the human auditory system. Furthermore, we show that the learning rate is a crucial design parameter even for adaptive gradient-based optimizers, which has been generally overlooked in the literature. Also, we found that waveform matching performance metrics must be used with caution as they in certain situations can fail completely. Finally, we show that a loss function based on scale-invariant signal-to-distortion ratio (SI-SDR) achieves good general performance across a range of popular speech enhancement evaluation metrics, which suggests that SI-SDR is a good candidate as a general-purpose loss function for speech enhancement systems.

Time-domain speech enhancement using generative adversarial networks

Speech enhancement improves recorded voice utterances to eliminate noise that might be impeding their intelligibility or compromising their quality. Typical speech enhancement systems are based on regression approaches that subtract noise or predict clean signals. Most of them do not operate directly on waveforms. In this work, we propose a generative approach to regenerate corrupted signals into a clean version by using generative adversarial networks on the raw signal. We also explore several variations of the proposed system, obtaining insights into proper architectural choices for an adversarially trained, convolutional autoencoder applied to speech. We conduct both objective and subjective evaluations to assess the performance of the proposed method. The former helps us choose among variations and better tune hyperparameters, while the latter is used in a listening experiment with 42 subjects, confirming the effectiveness of the approach in the real world. We also demonstrate the applicability of the approach for more generalized speech enhancement, where we have to regenerate voices from whispered signals.

EMD BASED SPEECH ENHANCEMENT USING SOFT AND HARD THRESHOLD TECHNIQUES

In last few decades many attempts have been made on speech signals to eliminates the noise. Purpose of use any speech enhancement algorithm is to eliminate noises in variety of environments; most prominent of which are telecommunication applications. The purpose of this paper is to development of a novel speech enhancement algorithm which offers superior noise reduction over current methods. This Research paper work demonstrates a novel time domain speech enhancement algorithm for speech signals called empirical mode decomposition (EMD). EMD decomposes the speech signal corrupted by noise signal into a finite number of band limited signals known as intrinsic mode functions (IMFs), using iterative procedure called sifting process. These IMFs are denoised by using two different techniques first method is IMFs thresholding method or direct method of speech enhancement and another technique is IMF frame based method are discussed in this work. Both the methods use soft and hard threshold techniques for denoising the IMFs which are obtained from EMD. These algorithms are implemented empirically by using MATLAB software on real time speech data. Experimental Results shows that IMFs frame based method superior than direct method this can be tested by adding noises with the different SNR values to the clean speech.

Speech Enhancement with Nonstationary Acoustic Noise Detection in Time Domain

This letter proposes a new time domain speech enhancement technique for signals corrupted by nonstationary acoustic noises. In this method, the noise components are detected and attenuated directly from the corrupted speech samples. They are obtained with a robust estimation of the noise standard deviation considering any speech and noise amplitude distribution. These values are used to define a noise selection threshold. Additionally, this solution does not require the usage of any spectral analysis or temporal decomposition as a pre-processing phase. The experiments results show that the proposed scheme leads to significant improvement in the speech quality and intelligibility when compared to competing enhancement approaches.